{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/bitkira/Colab/blob/main/tutorial_notebooks_zh/benchmark_and_evaluation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install git+https://github.com/EvoAgentX/EvoAgentX.git"
      ],
      "metadata": {
        "id": "Hkx1jKIicKSm"
      },
      "id": "Hkx1jKIicKSm",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install PyPDF2 selenium html2text fastmcp"
      ],
      "metadata": {
        "id": "002BvH9xcK5g"
      },
      "id": "002BvH9xcK5g",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "a24583c0",
      "metadata": {
        "id": "a24583c0"
      },
      "source": [
        "# 基准测试和评估教程\n",
        "\n",
        "本教程将指导你使用 EvoAgentX 设置和运行基准测试评估。我们将使用 HotpotQA 数据集作为示例，演示如何设置和运行评估过程。\n",
        "\n",
        "## 1. 概述\n",
        "\n",
        "EvoAgentX 提供了一个灵活且模块化的评估框架，使你能够：\n",
        "\n",
        "- 加载和使用预定义的基准数据集\n",
        "- 自定义数据加载、处理和后期处理逻辑\n",
        "- 评估多代理工作流的性能\n",
        "- 并行处理多个评估任务\n",
        "\n",
        "## 2. 设置基准测试\n",
        "\n",
        "首先，你需要导入相关模块并设置评估过程中代理将使用的语言模型（LLM）。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1a38650c",
      "metadata": {
        "id": "1a38650c"
      },
      "outputs": [],
      "source": [
        "from evoagentx.config import Config\n",
        "from evoagentx.models import OpenAILLMConfig, OpenAILLM\n",
        "from evoagentx.benchmark import HotPotQA\n",
        "from evoagentx.workflow import QAActionGraph\n",
        "from evoagentx.evaluators import Evaluator\n",
        "from evoagentx.core.callbacks import suppress_logger_info"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "04485725",
      "metadata": {
        "id": "04485725"
      },
      "source": [
        "\n",
        "### 配置 LLM 模型\n",
        "你需要一个有效的 OpenAI API 密钥来初始化 LLM。建议将 API 密钥保存在 `.env` 文件中，并使用 `load_dotenv` 函数加载它："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bb385e6c",
      "metadata": {
        "id": "bb385e6c"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from dotenv import load_dotenv\n",
        "\n",
        "try:\n",
        "    from google.colab import userdata\n",
        "    OPENAI_API_KEY = userdata.get(\"OPENAI_API_KEY\")\n",
        "except ImportError:\n",
        "    OPENAI_API_KEY = None\n",
        "\n",
        "if not OPENAI_API_KEY:\n",
        "    load_dotenv()\n",
        "    OPENAI_API_KEY = os.getenv(\"OPENAI_API_KEY\")\n",
        "\n",
        "llm_config = OpenAILLMConfig(model=\"gpt-4o-mini\", openai_key=OPENAI_API_KEY)\n",
        "llm = OpenAILLM(config=llm_config)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6d1de1f2",
      "metadata": {
        "id": "6d1de1f2"
      },
      "source": [
        "\n",
        "## 3. 初始化基准测试\n",
        "EvoAgentX 包含多个预定义的基准测试，用于问答、数学和编码等任务。有关现有基准测试的更多详细信息，请参阅 [Benchmark README](https://github.com/EvoAgentX/EvoAgentX/blob/main/evoagentx/benchmark/README.md)。你还可以通过扩展基础 `Benchmark` 接口来定义自己的基准测试类，我们在 [自定义基准测试](#custom-benchmark) 部分提供了一个示例。\n",
        "\n",
        "在这个示例中，我们将使用 `HotpotQA` 基准测试。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "21a26d9a",
      "metadata": {
        "id": "21a26d9a"
      },
      "outputs": [],
      "source": [
        "benchmark = HotPotQA(mode=\"dev\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a4fc5248",
      "metadata": {
        "id": "a4fc5248"
      },
      "source": [
        "其中 `mode` 参数决定加载数据集的哪个部分。选项包括：\n",
        "\n",
        "* `\"train\"`：训练数据\n",
        "* `\"dev\"`：开发/验证数据\n",
        "* `\"test\"`：测试数据\n",
        "* `\"all\"`（默认）：加载整个数据集\n",
        "\n",
        "数据将自动下载到默认缓存文件夹，但你可以通过指定 `path` 参数来更改此位置。\n",
        "\n",
        "## 4. 运行评估\n",
        "一旦你准备好了基准测试和 LLM，下一步就是定义你的代理工作流和评估逻辑。EvoAgentX 支持完全自定义基准测试示例的处理方式和输出的解释方式。\n",
        "\n",
        "以下是如何使用 `HotpotQA` 基准测试和 QA 工作流运行评估。\n",
        "\n",
        "### 步骤 1：定义代理工作流\n",
        "你可以使用预定义的工作流之一或实现自己的工作流。在这个示例中，我们使用为问答设计的 [`QAActionGraph`](https://github.com/EvoAgentX/EvoAgentX/blob/main/evoagentx/workflow/action_graph.py#L99)，它简单地使用自一致性来生成最终答案：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2654dd40",
      "metadata": {
        "id": "2654dd40"
      },
      "outputs": [],
      "source": [
        "workflow = QAActionGraph(\n",
        "    llm_config=llm_config,\n",
        "    description=\"This workflow aims to address multi-hop QA tasks.\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0a817bd5",
      "metadata": {
        "id": "0a817bd5"
      },
      "source": [
        "\n",
        "### 步骤 2：自定义数据预处理和后处理\n",
        "\n",
        "评估的下一个关键方面是正确地在基准测试、工作流和评估指标之间转换数据。\n",
        "\n",
        "### 为什么需要预处理和后处理\n",
        "\n",
        "在 EvoAgentX 中，**预处理**和**后处理**是确保基准测试数据、工作流和评估逻辑之间顺畅交互的重要步骤：\n",
        "\n",
        "- **预处理（`collate_func`）**：  \n",
        "\n",
        "    来自 HotpotQA 等基准测试的原始示例通常包含结构化字段，如问题、答案和上下文。但是，你的代理工作流通常期望一个单一的提示字符串或其他结构化输入。`collate_func` 用于将每个原始示例转换为你的（自定义）工作流可以使用的格式。\n",
        "\n",
        "- **后处理（`output_postprocess_func`）**：\n",
        "\n",
        "    工作流输出可能包括推理步骤或超出最终答案的额外格式。由于 `Evaluator` 内部调用基准测试的 `evaluate` 方法来计算指标（例如，精确匹配或 F1），通常需要以干净的格式提取最终答案。`output_postprocess_func` 处理这一点，确保输出适合评估。\n",
        "\n",
        "简而言之，**预处理为工作流准备基准测试示例**，而**后处理为评估准备工作流输出**。\n",
        "\n",
        "在以下示例中，我们定义了一个 `collate_func` 来将原始示例格式化为工作流的提示，以及一个 `output_postprocess_func` 来从工作流输出中提取最终答案。\n",
        "\n",
        "可以使用 `collate_func` 格式化基准测试中的每个示例，它将原始示例转换为代理的提示或结构化输入。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "09b31226",
      "metadata": {
        "id": "09b31226"
      },
      "outputs": [],
      "source": [
        "def collate_func(example: dict) -> dict:\n",
        "    \"\"\"\n",
        "    Args:\n",
        "        example (dict): A dictionary containing the raw example data.\n",
        "\n",
        "    Returns:\n",
        "        The expected input for the (custom) workflow.\n",
        "    \"\"\"\n",
        "    problem = \"Question: {}\\n\\n\".format(example[\"question\"])\n",
        "    context_list = []\n",
        "    for item in example[\"context\"]:\n",
        "        context = \"Title: {}\\nText: {}\".format(item[0], \" \".join([t.strip() for t in item[1]]))\n",
        "        context_list.append(context)\n",
        "    context = \"\\n\\n\".join(context_list)\n",
        "    problem += \"Context: {}\\n\\n\".format(context)\n",
        "    problem += \"Answer:\"\n",
        "    return {\"problem\": problem}"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5a962d9a",
      "metadata": {
        "id": "5a962d9a"
      },
      "source": [
        "\n",
        "在代理生成输出后，你可以定义如何使用 `output_postprocess_func` 提取最终答案。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4079b836",
      "metadata": {
        "id": "4079b836"
      },
      "outputs": [],
      "source": [
        "def output_postprocess_func(output: dict) -> dict:\n",
        "    \"\"\"\n",
        "    Args:\n",
        "        output (dict): The output from the workflow.\n",
        "\n",
        "    Returns:\n",
        "        The processed output that can be used to compute the metrics. The output will be directly passed to the benchmark's `evaluate` method.\n",
        "    \"\"\"\n",
        "    return output[\"answer\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2164ef6c",
      "metadata": {
        "id": "2164ef6c"
      },
      "source": [
        "\n",
        "### 步骤 3：初始化评估器\n",
        "评估器将所有内容联系在一起——它在基准测试上运行工作流并计算性能指标。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4f8aaae1",
      "metadata": {
        "id": "4f8aaae1"
      },
      "outputs": [],
      "source": [
        "evaluator = Evaluator(\n",
        "    llm=llm,\n",
        "    collate_func=collate_func,\n",
        "    output_postprocess_func=output_postprocess_func,\n",
        "    verbose=True,\n",
        "    num_workers=3\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f7898e8c",
      "metadata": {
        "id": "f7898e8c"
      },
      "source": [
        "如果 `num_workers` 大于 1，评估将在多个线程上并行进行。\n",
        "\n",
        "### 步骤 4：运行评估\n",
        "现在，你可以通过向评估器提供工作流和基准测试来运行评估：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1ffeaead",
      "metadata": {
        "id": "1ffeaead"
      },
      "outputs": [],
      "source": [
        "with suppress_logger_info():\n",
        "    results = evaluator.evaluate(\n",
        "        graph=workflow,\n",
        "        benchmark=benchmark,\n",
        "        eval_mode=\"dev\", # Evaluation split: train / dev / test\n",
        "        sample_k=10 # If set, randomly sample k examples from the benchmark for evaluation\n",
        "    )\n",
        "\n",
        "print(\"Evaluation metrics: \", results)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "258182cb",
      "metadata": {
        "id": "258182cb"
      },
      "source": [
        "其中 `suppress_logger_info` 用于抑制日志信息。\n",
        "\n",
        "有关完整示例，请参考 [基准测试和评估示例](https://github.com/EvoAgentX/EvoAgentX/blob/main/examples/benchmark_and_evaluation.py)。\n",
        "\n",
        "\n",
        "## 自定义基准测试\n",
        "\n",
        "要定义自定义基准测试，你需要扩展 `Benchmark` 类并实现以下方法：\n",
        "\n",
        "- `_load_data(self)`：\n",
        "    \n",
        "    加载基准测试数据，并设置 `self._train_data`、`self._dev_data` 和 `self._test_data` 属性。\n",
        "\n",
        "- `_get_id(self, example: Any) -> Any`：\n",
        "\n",
        "    返回示例的唯一标识符。\n",
        "\n",
        "- `_get_label(self, example: Any) -> Any`：\n",
        "\n",
        "    返回与给定示例关联的标签或真实值。\n",
        "\n",
        "    这用于在评估期间将预测与正确答案进行比较。输出将直接传递给 `evaluate` 方法。\n",
        "\n",
        "- `evaluate(self, prediction: Any, label: Any) -> dict`：\n",
        "\n",
        "    基于预测和真实标签（从 `_get_label` 获取）计算单个示例的评估指标。\n",
        "    此方法应返回指标名称和值的字典。\n",
        "\n",
        "- `evaluate(self, prediction: Any, label: Any) -> dict`:\n",
        "\n",
        "    计算单个示例的评估指标，基于其预测和真实标签（从 `_get_label` 中获取）。\n",
        "    此方法应返回一个包含指标名称和值的字典。\n",
        "    \n",
        "有关基准实现的完整示例，请参阅 [HotPotQA](https://github.com/EvoAgentX/EvoAgentX/blob/main/evoagentx/benchmark/hotpotqa.py#L23) 类。"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}